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Abstract. Probabilistic Computation Tree Logic (PCTL) is a well-known modal logic 
which has become a standard for expressing temporal properties of finite-state Markov 
chains in the context of automated model checking. In this paper, we give a definition 
of PCTL for noncountabie-space Markov chains, and we show that there is a substan- 
tial affinity between certain of its operators and problems of Dynamic Programming. 
After proving some uniqueness properties of the solutions to the latter, we conclude 
the paper with two examples to show that some recovery strategies in practical appli- 
cations, which are naturally stated as reach-avoid problems, can be actually viewed as 
particular cases of PCTL formulas. 



1. Introduction 

Reachability analysis of deterministic dynamical systems constitutes a practically 
important and intensely researched area in control theory. Over the years, a wide 
variety of tools and methods have been developed to verify the dynamic properties 
of these systems, for examples see [28, 10, 1, 2, 3, 29]. In particular, in [28, 29] 
the reachability problems considered are solved via dynamic programming (DP). As 
a result, a large number of exact and approximate methods for solving the central 
Bellman equation in DP [6, 7, 5, 34] can be exploited for the solution of verification 
problems of deterministic dynamical systems. 

Recently, reachability analysis of stochastic Markovian processes has gained signifi- 
cant interest, and mechanisms for the verification of safety and performance properties 
by means of a control policy have been explored. An example of such a problem is to 
find the probability starting from a certain state x, of reaching a "safe" set within a 
certain number of time-steps, where the state x could be labelled "almost safe" if such 
probability is greater than, say, 1 — e. A related problem, which has been studied re- 
cently by some of the authors, is that of maximizing the probability of reaching a "safe" 
set, while avoiding a "bad" set [38, 13]. This problem arose as a remedy for the im- 
possibility of imposing hard state constraints in stochastic model predictive control. 
In general, if one considers an infinite trajectory of a stochastic system, every com- 
pact state-constraint set is going to be violated almost surely at some time. Thus, a 
good course of action when this happens is to devise a recovery strategy to drive the 
controlled system from the "unsafe" states back to the set of "safe" states. 

If a control variable is unavailable or a control policy has been predetermined, the 
verification of the stochastic system reduces to calculating the likelihood of the occu- 
rence of certain events. In this manner, the above problem is directly related to sto- 
chastic model checking of finite-state Markov models in that the analysis involves both 
reachability and likelihood computations. Therefore, it is reasonable to consider an 
extension of Probabilistic Computation Tree Logic (PCTL), a modal logic developed for 
finite-state Markov chains, which forms the foundation for the automated verification 
tools for finite -state Markov models, to general state-space Markov chains. 
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Algorithms for stochastic model checking finite-state Markov models come from 
standard deterministic model checking, linear algebra, and the analysis of Markov 
chains. Finite state model checkers include the software tools PRISM [22], SMART 
[14], E N MC 2 [20], and MRMC [24], and have been used to solve various problems 
over the last few years. In the area of systems biology, probabilistic model checking has 
been used in the analysis of biological pathways [19, 26] and signalling events [31]. 
Additional examples of the use of stochastic model checking include the probabilistic 
verification of security protocols [4], dynamic power management [37], and residual 
risks in safety-critical systems [17]. 

In this paper, we consider the verification of general state-space Markov chains 
through an extension of the standard grammar and semantics of PCTL to non-countable- 
state Markov chains (the reader can find a similar extension in [23]). As with the 
finite case, the evaluation of a PCTL formula can be recursively reduced to the truth 
of atomic propositions by employing computations dictated by the PCTL semantics. 
In this process of reduction, certain rules of the semantics simply stipulate unions or 
intersections of sets, while others involve the computation of integrals. It is in the com- 
putation of the integrals where the bulk of the algorithmic methodology is contained. 
We show that the "bounded until" operator, which considers the property of hitting 
a "safe" from an "unsafe" set over a finite time horizon, can be evaluated through a 
dynamic recursion. Additionally, we prove that the "unbounded until" operator, which 
considers the property of hitting a "safe" set from an "unsafe" set at some point in time, 
can be evaluated via a DP-like Bellman equation. Further, we emphasize that, while in 
the numerical examples provided we grid the state space in order to solve the integral 
equations, any method in the literature for the numerical computation of a DP can be 
exploited for this problem. 

Outline of the work: In section 2 we review the standard grammar and semantics of 
PCTL for finite-state Markov chains. In section 3 we extend the grammar and semantics 
of PCTL to general state-space Markov chains. The uniqueness of a certain function 
associated with the "unbounded until" property is considered in section 4. Finally, 
section 5 concludes the paper with some applications and numerical examples. 

2. Probabilistic Computation Tree Logic 

In this section we quickly review the definition and semantics of PCTL for finite-state 
Markov chains. The reader is referred to the original paper [18] or to the excellent sur- 
vey [27] for a detailed exposition. 

2.1. Labelled Markov chains. 

Definition 1. A homogeneous, discrete-time, finite-state Markov Chain is a triple {X, x, Q), 
where: 

o X is a finite set of states; 
o x is the initial state; 

o Q is a transition probability matrix, which assigns to each pair of states (x^Xj) the 
probability Q XjjX| of going from the state x,- to the state Xj at a given time. 

Consider the sample space f2 := O^o^-' containing the possible trajectories co = 
(x , x 1} x t , ...) of the chain, and the product cr-algebra Jonn. For a given trajectory 
co = (_x ,x 1 , ...,x t , ...), let co[t) := x t . It can be shown [9, pp. 90-91] that there exists 
a unique probability measure on denoted by P X G), such that ¥ X (X = x) = 1 

and P x (X t+1 = x t+1 | X t = x t ,X t _ 1 = x t _ v ...,X = x) = Q(x t , x t+1 ). 

Definition 2. Let j4 be a finite set of atomic propositions. A labelled Markov Chain is 
a quadruple (X,x,Q, L), where: 
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o (X,x,Q) is a finite-state Markov chain; 

o L : X —> 2^ is a set-valued function that assigns to each state xel the set L(x) c j4 
of all those atomic propositions that are true in the state. 

2.2. Grammar and semantics of PCTL. The grammar of PCTL is as follows: 

o T is a formula (meaning "true"). 

o Each atomic proposition in A e j4 is a formula. 

o If $ and * are formulas, then -i$ and $ A * are formulas. 

o If cj) is a "path formula" (see below) and p e [0, 1], then P^ p [0] is a {state) formula. 
Here and throughout the rest of the paper, ~ is just shortand for one of the relations 
<, ^, >, or ^. For example, P^ 09 [<£] is one such formula, where "~" = and 
p = 0.9. 

The above grammar defines state formulas, that is, formulas whose truth can be de- 
cided for each state x e X. The meaning of the formulas in the first three points is 
the usual one in the standard logic of propositions. The other standard formulas and 
operators can be obtained by means of combinations of the above ones. For example, 
F ("false") can be defined as -iT, $ V * ("inclusive or") as A and 4> — » * 
(formal implication) as -i$ V 

The last kind of formula is what makes PCTL a modal logic, since it allows to express 
the fact that, with probability contained in some range, something will happen in time. 
It relies on the definition of path formulas, that is, formulas whose truth is decided 
for paths co e £1. A formula like P i09 [<£] means, intuitively, that the probability of 
taking a path that satisfies cf> is at least 0.9. If 4> and * are state formulas, we define 
the following to be path formulas: 
o f$ ("next"); 
o $ q/^ k * ("bounded until"); 
o $ °U * ("unbounded until"). 

Intuitively, X§ means that next state will satisfy <3>; 3> °U^ k * means that at some 
time i, within fc steps, * will become true, and until that time <3> will remain true; and 
4> °U * means that at some arbitrarily large time i, * will become true, 4> being true 
until then. (See the semantics below for a precise definition.) 



For example, the statement x 1= P^ Q9 *J means: With probability at least 

0.9, starting from the state x, within 10 steps * will become true, and until then $ 
will remain true. In a sense, the formula P^ 09 [<$ W^ 10 itself denotes the set of 
all states E, such that, starting from £, with probability at least 0.9, etc. The above 
statement is of course equivalent to x being a member of such a set. 

The two "until" operators allow us to define other operators which are standard in 
any temporal logic. For example, given a state formula 4>, the path formula 4>, 
which means that eventually, within k steps, 4> will happen, can be defined as T °l/^ k 4>, 
and the path formula <> 4>, which means that eventually, at some time, $ wiZZ happen, 
can be defined as T ^ 4>. Formulas containing the standard "always" operator □ can 
also be defined, although not in the straightforward way one may expect at first sight: 
□<3> := is not a correct definition, since PCTL does not allow for the negation of 

path formulas. See [27] for details. 

Let A denote an atomic proposition, $ and * denote two state formulas, and <fi 
denote a path formula. The semantics of PCTL is defined as follows: 



xl=T for all xeX 



x 1= -i$ <=> x 4> 



x 1= <£ A * <=> xl=<I> and x 1= * 



x^ p [0] <=> P x ({</>}) ~P 
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With loose notation, {<fi} stands for the set of all the paths co that satisfy a given 
path formula c/>. Here is the related semantics: 

co 1= %$^>co{l) 1= $ 

(2.1) co 1= $ *<=>3i s= k : 1= * and Vj < i, co{j) 1= $ 

co 1= $ ^ *<=>3i e N : co(i) 1= * and Vj < i, co(j) N * 

Due to the latter definitions, if c6 is a path formula then {c/>} = {co:col=c/>}is always 
an event, that is, it always belongs to J. 

The great relevance of PCTL for finite Markov chains lies, above all, in the fact that 
the validity of arbitrarily complex formulas at a given state can be decided exactly and 
in finite time. In particular, dealing with the common operators -i, A, V etc. requires 
just the parsing of a tree of sub-formulas; a "bounded until" formula can be decided 
recursively; and an "unbounded until" formula requires the solution of a system of 
linear equations. For these matters the reader is referred to [18] and [27]. We shall 
not delve into details here, because the relatively easy methods available for finite 
Markov chains cannot be easily extended to the case of noncountable-space Markov 
processes, with respect to which the decision of PCTL formulas will be a matter of 
computing integrals recursively, or solving integral equations. 

3. PCTL FOR GENERAL MARKOV PROCESSES 

In what follows we define PCTL grammar and semantics on a noncountable space X 
in terms of a stochastic kernel Q and a probability measure P x defined on the space of 
trajectories of the process. The reader is also referred to [23] for an abstract extension 
of PCTL to general Markov chains. 

Given a nonempty Borel set X (i.e., a Borel subset of a Polish space), its Borel 
cr-algebra is denoted by By convention, when referring to sets or functions, 

"measurable" means "Borel-measurable." If X is a nonempty Borel space, a stochastic 
kernel on X is a map Q : X x — » [0, 1] such that Q(x, •) is a probability measure 
on X for each fixed x e X, and Q(-,B) is a measurable function on X for each fixed 
B e <B(X). 

Let X be a nonempty Borel set, and let Q(-,-) be a stochastic kernel on X. For 
each t = 0, 1, . . . , we define the space H t of admissible histories up to time t as H t := 
n,=o^- ' t e N . A generic element h t of H t , called an admissible t -history is a vector 
of the form h t = (x , x 1 ,..., x t ), with Xj eX for j = 0, . . . , t. Hereafter we let the 
cr-algebra generated by the history h t be denoted by J t , t e N . Suppose the initial 
state x is given, and let 5 X denote the Dirac measure at {x}. We consider the canonical 
sample space Q := O^o^ an< ^ t ^ ie P r °duct cr-algebra £ on £1. By a standard result 
of Ionescu-Tulcea [35, Chapter 4, §3, Theorem 5] there exists a unique probability 
measure, denoted by P x (-) on the measurable space (f2,£) such that P X (X eB) = 
5 X (B) and P x (X t+1 eB | h t ) = Q(x t ,B) forB e 

3.1. Grammar and semantics. The "labelling" function L is introduced in [18] and 
[27] as a means to specify which states satisfy which atomic propositions. In other 
words, it is just a particular way to look at the relation "x satisfies A". It should be 
clear that an equally legitimate way to accomplish the same is to substitute from the 
beginning the "labelling" function L : X — » 2^ with a function S : j4 — » 2 X , that 
assigns to each atomic proposition A the set S(A) of all those states that satisfy A. The 
semantics can be redefined accordingly in a straightforward way: 

<=> xeS(A) 

But since there is no substantial difference between saying that a state x satisfies a 
given property, and stating that x belongs to a set, namely the set of all the states 
that satisfy that property, it is easily seen that proceeding along this way one may 
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drop tout-court the distinction between formulas and sets of states satisfying them. 
In the following, we shall follow this idea consistently (mainly for ease of notation). 
Thus, from now on, we shall assume that the properties expressed by formulas are 
actually encoded by measurable sets $ c X, we will use the letters A, 4>, ... for both 
the formulas (or atomic propositions) and the sets that encode them, and we will use 
the notations x 1= <£ and x e <£ somewhat interchangeably. In the same fashion, we 
will drop the distinction between path formulas and events in the process's probability 
space. 

Let us denote the family of atomic propositions with a family of Borel measurable 
sets j4 c 25(X), where X e j4 '. The grammar of PCTL is defined exactly as before: 
o T is a formula (encoded by the whole space X) . 
o Each atomic proposition Aej/isa formula, 
o If 4> and * are formulas, then -1$ and <3> A ^ are formulas, 
o If cp is a path formula and p e [0, 1], then P^ p [0] is a {state) formula. 
The following are path formulas: $ *, and $ ^ 

Now we define the semantics of PCTL formulas for each possible initial state x e X. 
Let A denote an atomic proposition and $ and * denote formulas (measurable sets). 
We define: 

xl=T for all xeX 
x\=A <=> xeA 
x 1= -i$ <=> x e * c 
x 1= $ A * <=> xe$n* 
xNP„ p |>] <=> P x ({<£}) ~p 



As in the finite state case, we can also define F := -iT, 4> V * := A and 
4> — » * := V and of course we have 

x 1= $ V * <=> xe*U* 

xl=$— <=> xe$ c u* 

Note that all the formulas obtainable from atomic propositions by means of the op- 
erators -i, A, V,— » are encoded by sets that belong to ct(j</). The semantics of path 
formulas is defined exactly as in equation (2.1). 

3.2. "Next". We will now examine the state formulas derived from the three path 
formulas in greater detail. The formula arising from the "next" operator is trivial. 
Indeed, 

F x (ar*) = P x ({<o : «o(i) e *}) = P x (X x e *) = Q(x, $) 

Hence, 

x 1= P^ p [SF$] <=> Q(x, <t>) ~ p 

Note that P^ p [%$~\ is a measurable set in its own right. For example, P^ 5 [5T4>] is 
the 0.5-sub-level set of the measurable function Q(-,4>). Indeed, for each $ e 23QO, 
the set {x : Q(x, 4>) ~ p} belongs to by the measurability of Q(-, 

3.3. "Bounded until". Suppose that the process starts from x = x. On the probabil- 
ity space of our Markov process we define the following event: 

j$ q/^ k := {x e u {x e $,x 1 e *}U 

j {x,x 1 e$,x 2 e*}u...U{x,x 1 ,...,x fc _ 1 e$,x fc e*} 

= {x e u {x e 4>\*,x! e u {x,x 1 e $\*,x 2 e *} 

u ... u {x,^,.,.,*^ e $\*,x fc e 
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where U denotes a disjoint union. The probability of the set {$ °U^ k can be com- 
puted directly using the additivity of ¥ x : 



¥ x (* "U^ 



(3.2) 



€ *) + ¥ x ( Xj e *\*. x 2 e *) + • • • 
• • • + P x (x l5 ...,x k _ t e <*>\*, x k e *) 







if x e * 
if x e 

otherwise 



By the Markov property, all the latter probabilities can be expressed in terms of Q. For 
instance: 

¥ x (i 1 ,..,xne$\*,x t e*) 



4>\* 



Q(? fc _ 3 ,d? fc _ 2 )- 



Q(^ 2 ,d£ k _i)Q(^i,*). 



4>\* 



Nevertheless, P x (<I> ^/^ k can be computed more expressively in a recursive 
fashion. Let ^# 6 (X) be the set of all the measurable and bounded functions defined 
over X. Jl h (X~) is a Banach space with the norm \f\ tx '■= sup xeX /(x). L et tne opera- 
tor L : ^t b (X) — » Ji h iX) be defined as follows: 



(3.3) 



L[W](x) := l*(x)+ Wx) 



Q(x,d£)W(£) 



Given $ and let „^° 1 (<I>,']>) c Jt h {X) be the set of functions W such that: 
o forallxeX, ^ W(x) ^ 1; 
o for all x e W(x) = 1; 
o for all x e u *), W(x) = 0. 

Lemma 3. The set Ji^{§, is closed in Ji h (X)> an d L maps M^ifi, into itself. 
Proof. The closedness of b 

are preserved even by pointwise convergence. Let W ^ . b 
of L [W] follows from the fact that if Q is a stochastic kernel, and W is a measurable 
bounded function, then the function x — > f Q(x, d£)W(n is also measurable and 
bounded (see for instance [21, Appendix C]). The bounds ^ L[W](x) ^ 1 are obvi- 
ous, since the same bounds hold for the integral, Q(x, •) being a probability onX. The 
fact that L[W](x) = 1 Vi e * and L[W](x) = Vx el\($ U *) is also obvious due 
to the indicator functions in the definition of L. □ 



01 ($, is trivial, because all of its three defining property 

01 (4>, The measurability 



(3.4) 



For fixed $ and let us now define recursively: 

V„ := 1* 
V k+1 :=L[V k ] 



Lemma 4. For all k ^ 0, V k [x) = V x °U^ k Moreover, for all x, the sequence 
{V;.(x)} is nondecreasing. 

Proof. Substituting recursively V 1 into V 2 , V 2 into V 3 and so on, we obtain 
V 2 (x) = + l$\*(x)Q(x, 



+ 1 



4>\* 



(x) 



Q(x,d^)Q(?i,*), 



4>\* 



V 3 (x) = l*(x) + W(x)Q(x, 
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4>\* 



4>\* 



Q(?i,d? 2 X2(?2,*) 



i— 1 times 



QCjr.dgJ--- 



Q(? i -2,d? i -iX2(? i -i,*). 



Then, by the Markov property, 

k 

VJt(x) = l*(x) + 1#\*(x)5] f *(*i.-.*j-i e $ \*> x i e *) 



1=1 



= P x (x e *) + J P x (x, x u x t _i e *\*, x, e *) 

i=l 

= P x ({x e *} u ... u {x,x 1; ....Xfc.! e <i>\*,x fc e *}) 
= P x ($ 1^* *) 

The first assertion is proved. The second one is easily proved by induction. Obviously 
Vj(x) - V (x) = l $ ^(x)Q(x,*) ^ 0. Suppose now that V k+1 (x) - V fc (x) ^ 0. Then 

W x )~ V fc+1 (x) = U\v(.x) I x Q(x, d?) (Vfc+i(?) - V k (0) 0. It follows by induction 
that for all k ^ and all x e X we have V fc+1 (x) 3? V fc (x). □ 

The semantics of the "bounded until" PCTL operator is now easy to explain. In view 
of Lemma 4, given $ and * we have: 

x 1= P„ p [$ <5/ s * *] <=> V fc (x)~p 

Since V fc is Borel measurable, any super- or sub-level set of the kind P^ p ^/^ k #1 is 
a Borel subset of X. 

3.4. "Unbounded until". Finally, we develop the "unbounded until" PCTL formula 
in detail. Suppose, as before, that the process starts from x = x. In the process's 
probability space we consider the event 

{$ °U *} = {3t e N : x,x x ,...,x^ x e $,x T e *} 

(3.5) = {x e u ... u {x,x l3 ...,x k _ x e *, jcj e *} u ... 

= {x e u ... u {x,x l3 ...,x k _ x e <f>\V,x k e u ... 

Its probability is as follows: 

P x = 



(3.6) 



if x e 

p *( x i' x fc-i e x * e *) if x e $\#, 
otherwise. 



Notice, however, that the "unbounded until" event is indeed the limit of the nonde- 
creasing sequence of "bounded until" events we have considered above, i.e., 

+00 

{$ = [J {<J> 



k=0 
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Consequently, for all x its probability can be obtained as the following limit: 
P x O °U = lim V x f$ ^ k = lim V k {x) 

k— »+oo ^ k—>+oo 

(This limit is also a supremum, since the V k form a nondecreasing sequence.) We define 
(3.7) V(x):= lim V k (x) 

k—>-hoo 

Lemma 5. The function V defined in (3.7) belongs to ^° 1 (^,^) and satisfies the fol- 
lowing integral equation: 



(3.8) V(x) = l*(x) + l^M Q(x, d?)V(?) 

Jx 

(In other words, it is a fixed point for L.) 

Proof. The three properties required for the belonging to M^(§, are immediate, 
for they hold for all the V k 's. Consider again the recursive definition (3.4): 

r 



(3.9) V k+1 (x) = 1*U) + 



From Lemmas 3 and 4, the V k s are Borel measurable and non-negative, and they 
form a nondecreasing sequence. By definition of V, they converge pointwise to V. 
Therefore, by the monotone convergence theorem (see for instance [35, Theorem 1, 
p. 13]) for all x we have 

Qfx,d£)V fc (?)= [ Q(x,dOV(0 

. X JX 

Hence, letting k —> +oo in both sides of (3.9), we obtain (3.8). □ 



lim 

k— »+oo 



The semantics of the "unbounded until" PCTL operator is now obvious. For given $ 
and we have: 

x 1= P^ p [$ °U <=> V(x) ~ p 

Since V is the limit of measurable functions, it is measurable itself, hence its super- or 
sub-level sets P^ p [4> W are again Borel subsets of X. 

3.5. Notes on equation (3.8). First of all, note that the function V defined in (3.7) is 
indeed a solution to equation (3.8), but it is by no means guaranteed to be its unique 
solution. As a counterexample, let us consider the operator we have mentioned in 
the finite case. Let * be a formula (set). The path formula 0* ("eventually *") is 
defined as T "It *. Its probability V*(x) = P x (T must therefore satisfy: 

(3.10) V(x)= l w (x) + l»c(x) Q(x,d£M£) 

Jx 

Suppose that the set * c is absorbing (that is, Q(x,*) = for all x e * c ). Then, 
it is easy to see that both V(x) = l^Cx) and V*(x) = 1 are solutions of (3.10) (the 
meaningful one being the former). As another limit example, consider the event OF 
("eventually, false will hold true"!). Its probability, both by immediate intuition and by 
calculation, must be zero for all x. Nevertheless, any constant function V is a solution 
to the corresponding equation: 



r 



V(x) = l (x) + l x (x) 



r 



Q(x,d£)V(0 

x 



Q(X,d£M£): 

Jx 

(irrespective of the structure of Q) . 

We can get around this issue with a characterization of V among the solutions of 
(3.8). We have the following result: 
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Lemma 6. Let {W a } be the family of all the non-negative solutions to (3.8), i.e., 



Wjx) = l^fx) + l^Qc) 



Then, for all x 



V(x) = infW a (x) = min W a (x) 



Proof. First, we show that, for any V k defined in (3.4) and for any non-negative solution 
W to (3.8), it holds V fc (x) sS W(x). Define V.^x) = onX. Then we have I[V_j] = 
V . Now, for all x e X, W(x) - Vl^x) = W(x) ^ by hypothesis. Assume that 
W(x) - V fc (x) ^ for all x. Then 

W{x) - V k+1 {x) = L[W](x) - L [V fc ](jc) 



^0. 



It follows by induction that V fc (x) ^ W[x) for all x e X and for all fc e N . 

Since the above inequality holds for all of the V k s, it also holds for their supremum 
V, that is, V(x) ^ W(x) for any non-negative solution W to (3.8). The assertion 
follows since, by Lemma 5, V is itself a solution to (3.8). □ 



4. Uniqueness of V 

This section treats the issue of uniqueness of solutions to the integral equation (3.8). 
We approach the problem from two different directions, the first is functional analytic: 



Proposition 7. Suppose that 



sup Q(x,$\*) < 1. 



Then 

(1) L is a contraction on ^t® 1 ^,^); 

(2) equation 3.8 has a unique solution V; 

(3) the elements V k defined in 3.4 converge to V in the 
X. 



norm, that is uniformly in 



Proof. Let a = sup xe< ^Q(x, $\*). Let W lt W 2 e ^ b 01 (<£,*)- For all x e * u 4> c 
|l[Wi](x) - i[W 2 ](x)| = 0, whereas for all x e <J>\*, we have 

|L[W 1 ](x)-L[W 2 ](x)| 



Qix.dOW^O 



Q(x,d?)W 2 (0 



Q(x,d£) IWaCO-WiCOl 
Q(x,d£) IWiCO-WaC^I 



4>\* 



Q(x,d£) p x -W 2 | 

4>\* 
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Since the above bound holds for each x, it holds also for the supremum over 4>\*, and 
consequently for the supremum overX: 

||l[Wi]-L[W 2 ]|| oo = sup|i[W 1 ](ac)-L[W 2 ]Cx)| 

This concludes the proof of claim (1). Claims (2) and (3) follow by the Contraction 
Mapping Theorem [36, Theorem 9.23] since Jt^ifi, *) is closed. □ 

Corollary 8. Suppose that sup^g^^ Q(x, $\*) < 1. Suppose moreover that Q satisfies 
the strong Feller or strong continuity [21, Appendix C] property. Then the restriction of 
V to 4>\* is continuous. 

Proof. Let V and V k denote the restriction to 4>\* of V and V k respectively. In particular, 
we have 



(4.1) 



V W = o 
Vfc+l(*) = 



Q(x,dOV fc (?) 



Obviously V is continuous. Due to the strong Feller property, x — > Q(x, is con- 
tinuous, and if V k is measurable then x — > J Q(x,d%)V k (£) and therefore V k+1 are 

continuous. By induction, all the V k are continuous. Hence, {V k } is a sequence of 
continuous functions that converges uniformly to V. Thus, V is also continuous. □ 



(4.2) 



The second direction is probabilistic: Let us define two random times 
t := inf{t e N \x t e *} and 
x' :=inf{t eN |x t eX\($U*)}. 

It is not difficult to see that t and t' are stopping times with respect to the filtration 
(5 r t)teN • Ak° observe that 

¥ x ($^*) = P x (t < t', t < oo), 



V(x) = P x (t < t', t < oo) = 



TAT 

s 

t=0 



Proposition 9. Assume that t A t' < oo almost surely. Then, for u e ^S^ 1 ^, *) we have 

(i) iz ^ V whenever u satisfies the functional inequality u ^ L [u], and 

(ii) u ^ V whenever u satisfies u^ L [u], 

where all inequalities are interpreted pointwise on X. In particular, V is the unique 
solution to the equation u = L [u] on the set M^i^, 

Proof. We prove (i) first. Fix u e ^#° 1 ($,*) and x e X. From Lemma 4 it follows 



readily that L is a monotone operator on 
times we arrive at 



■, . Iterating the inequality iz ^ L [u] n- 



uW^L[u](x) 



sSL[L[u]](x)^"- 



n— times 

^L[L[^i[u]"-]] 



4>U* 
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l.\*(C„-i) I Q(?„-i,d?>&)))) 



1*0) +W» Q( 



4>U* 



(*,d?i)(l* 



[ Q(?i, d? 2 ) f ... + .■ ■ f 1*(£„_ 2 )+ 

U\*ce„- 2 ) [ Q(^-2,(i?„-i)i*(?„-i}) 



+ l*\<pO) 



4>\* 



Q(?„-i,d?Ju(?„) 



= E V 



(n-l)ATAT' 

2 ^^t) 



t=0 



+ 



[l*\*P^(n-l)ATAT')(l*U* ' l, )(^nATAT')l{TAx'<oo}] • 

The left-hand side above is independent of n, and since t A t' < oo almost surely, 
taking limits we get 



uO) s= lim 



'(n— 1)AtAt' 
t=0 



+ 



J™, E x [l*\*(^(n-l)ATAT')- 

(1*U* ■ l, )(-^nATAT')l{xAT'<oo}] 



:E V 



t=0 



+ 



E x [1$\*G^tAt'X1$u* ■ u X^tAt')1{tAt'<oo}] 

= vO) + o. 

To justify the interchange of integration and limit above we have employed the mono- 
tone and the dominated convergence theorems for the first and the second terms, 
respectively, and since X xAT / ^ $ \ * by definition, the last expectation vanishes. Since 
u e ~M^(<&, *) and ieX are arbitrary, we see that u ^ L [u] implies u ^ V whenever 
u e ^? 2 ($, *). The proof of (ii) follows exactly the same arguments as above, with 
replacing every everywhere in the above steps; we omit the details. Unique- 
ness of V as a solution of the functional equation u = L[u] on the set Ji® x {§, *) 
follows at once from (i) and (ii). □ 



5. Examples 

We demonstrate the effectiveness of the PCTL verification methodology on two sim- 
ple problems with potentially important implications. The first example comes from 
the literature on fishery management, where multiple recovery strategies for a single 
species fishery are considered. The second example comes from the finance literature, 
where the problem of early retirement is explored. In both examples, the problems are 
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solved numerically by gridding the state space. It is of great interest to pursue more ef- 
fective and accurate solution methods for the DP integral equations using sophisticate 
methods commented on in the Introduction. 

5.1, Recovery Strategies in Fishery Management. Overexploitation can lead to both 
a decrease in the fish stock to a level below which maximum sustainable yield (MSY) 
cannot be supported and/or a decrease in fish stock to a level where net revenue has 
been driven to zero [15]. When the fish stock drops below this level, appropriate re- 
covery strategies are necessary to recover the fish stock while minimizing economic 
loss. In this example, we use the PCTL framework to evaluate the effectiveness of var- 
ious recovery strategies (or non-strategies) over a finite time horizon for the recovery 
of a fish population. 

We consider a discrete time Markov model of a single species fishery motivated by 
[33]. For a time horizon k = 0,1,... N, the evolution of the fish biomass within a 
fishable area is given by the stochastic difference equation [33] 

x k+1 = (1 - v k )x k + y k R[x k ) ~ S k C[x k ), 

where x k is the fish biomass at time k, R[-) is a function representing the recruitment 
(e.g., addition through birth) of fish, C(-) is the catch function, v k is a random variable 
that represents fish mortality during stage k, j k is a random variable representing 
the variability in the recruitment of the fish population, and 5 k is a random variable 
representing the variability in the catch. The species recruitment function is given by 



R(x fc ) = max jrx fc ^1- ,oj, 



where r e [0, 1] is the per-capita recruitment at time step k and K is equal to half the 
biomass limit (i.e., upper bound on the fish population) for the fishable area. 

We consider three different recovery strategies implemented through the target 
catch function. In the first, we apply a constant target catch according to the de- 
terministic MSY [25], i.e. 

K[r - ytf 
C[x kJ = C MSY = — , 

where fx is the deterministic mortality rate. The second recovery strategy is given by 
the Harvest Control Rule (HCR) 




MSY £ if x k < K, 



otherwise. 



Lastly, we consider the strategy C[x k ) = 0. 

Following [33], we assign the values K = 200, r = 1, and n = 0.2, and take all 
random variables to be i.i.d. according to the following distributions v ~ Jf[\i, 0.1 2 ), 
Y ~ jY{1,0.6 2 ), and 5 ~ J/' [1.1, 0.2 2 ). Using the MSY as a measure of safety for the 
system, we assign the target operating region for the fishery to be K = [150,400] and 
the safe operating region to be K' = ] 0,400]. 

For the verification of the control strategies, we consider the set of initial states (i.e., 
fish biomass at k = 0) that satisfy 

P >0 . 9 [K'V* S K]. 

That is, we are interested in the set of states that, with a probability greater than 90 
percent, will enter the target operating region K within JV = 5 time steps while remain- 
ing in K' until then. The functions satisfying the dynamic recursion (3.9) for the three 
different recovery strategies are shown in Figure 1. According to the computational 
results, the sets that satisfy the bounded until operator are approximately 0, [65,400], 
and [45,400] for the three policies respectively. It is interesting to note that under 
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50 100 150 50 100 150 



(a) Value Function (b) Value Function 




50 100 150 



(c) Value Function 

Figure 1. Results for the Recovery Problem at time k = 0. The func- 
tion V (-) for different recovery policies are given in (a) MSY, (b) 
HCR, and (c) Fishing Stop. 

the deterministic MSY quota policy the solution is the empty set, meaning that there 
are no initial states which result in recovery with 90 percent certainty over the short 
time horizon. Further, the gain in reliable recovery between the HCR strategy and a 
complete fishing stop is minimal, indicating that it may be in the economic interest of 
the fishery to use the HCR policy in the region. 

5.2. A Problem of Early Retirement. Recently, increased attention has been given 
to stochastic risk models with investment income in the discrete time setting [11, 16, 
40, 30, 39, 12, 41]. In most cases the probability of ruin over a finite or infinite time 
horizon is the main area of interest, with the infinite horizon case being mathematically 
easier and thus more popular in the literature [32]. Interestingly enough, personal 
retirement funds fall into the same category as basic ruin models, and therefore can 
be modeled as such. Further, the individual is often as concerned with the short term 
financial gain (e.g., achieving a financial target for the fund) as with the risk of losing 
the investment (i.e., ruin). 

Motivated by [8], we consider a discrete time Markov model of an individual re- 
tirement fund. Based on [32], the evolution of the retirement fund x k over a finite 
horizon k = 0, 1, . . . , JV is given according to the stochastic difference equation 

x k+1 = ax k (l + S k ) + bx k [l +R k ) + cx k + u k , 

where x k is the value of the retirement fund and u k is the yearly individual contribution 
to the fund. S k and R k are i.i.d. random variables representing the average rates of 
return for a safe investment and a risky investment over one year, a is the percentage 
of capital invested in the safe asset, b is the percentage of capital invested in the risky 
asset, and c is the percentage capital not invested at all. Note the restriction that 
a + b + c = 1. 
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For simplicity, all random variables are assumed to be i.i.d. with S k ~ ^(0.03, 0.005 2 ) 
and R k ~ jY{0.1, 0.2 2 ) for all k = 0, 1, . . . ,N. We consider three different investment 
strategies (i) a = 0.4, b = 0.4, and c = 0.2, (ii) a = 0.8, b = 0.2, and c = 0, and (iii) 
a = 0.2, b = 0.8, and c = 0. For each strategy, the yearly contribution is u k = 2500 for 
allfc = 0, 1,...,N. 

Consider the target set K = [200000, +oo[ and the safe set K' = ]0,+oo[. Over 
a finite time horizon of JV = 20 years, we would like to identify the set of all initial 
investments x e K such that the retirement fund hits the target set K (i.e., surpasses 
200000) while avoiding total financial ruin with a probability greater than 85 percent. 
To this end, we consider the PCTL formula 

(5.1) P^ 85 [K'^ 20 K]. 




0.5 1 1.5 2 % 1 ^ 

x x 10 s 

(a) Value Function (b) Value Function 




(c) Value Function 



Figure 2. The function V (-) for the Early Retirement problem at year 
k = for different investment policies are given in (a) a = 0.4, b = 
0.4, c = 0.2, (b) a = 0.8, b = 0.2, c = 0, and (c) a = 0.2, b = 
0.8, c = 0. 

For each investment strategy, the function satisfying the dynamic recursion (3.9) 
at time k = is shown in Figure 2. According to the computational results, the set 
that satisfies the bounded until operator for each strategy is given by (i) [70000, +oo[, 
(ii) [66500, +co[, and (iii) [51500, +oo[. Thus, with an initial investment of more 
than 51500 swiss francs, yearly contributions in the amount of 2500 swiss francs, and 
investment strategy (iii), an individual has an 85 percent chance of retiring within 20 
years. However, if we were to consider an increasing probability of success, at some 
point the strategy with the largest set satisfying the bounded until operator would 
switch from (iii) to (ii). 
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6. Conclusion and Future Work 

In this paper, we have extended the grammar and semantics of PCTL for finite- 
state Markov chains for the verification of general state-space Markov chains. We have 
shown that the bulk of the computational methodology is in the evaluation of the 
"bounded until" and "unbounded until" operators. And that the evaluation of these 
operators reduces to the computation of DP-like integral equations, for which there is 
a rich numerical history. 

In the future, extensions to the language to capture additional trajectories will be 
explored which maintain the DP-like structure. Also, numerical methods for the ef- 
ficient and accurate evaluation of the DP integral equations are being evaluated and 
applied to various sample problems. 
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